Three dimensional image fusion method and device and non-transitory computer-readable medium

ABSTRACT

Three dimensional image fusion method and device are disclosed. The method includes steps of obtaining a spherical image and a two dimensional image; projecting the two dimensional image onto a planar surface predetermined in a spherical coordinate system where the spherical image is located, so as to acquire a projected image having a predetermined number of feature points on the spherical surface corresponding to the spherical image; determining multiple mapping points on the spherical surface which correspond to the predetermined number of feature points, letting the predetermined number of feature points superpose on the multiple mapping points, respectively, and carrying out spatial coordinate conversion, so as to attain a converted image; and mapping multiple points except the predetermined number of feature points in the converted image onto the spherical image based on a predetermined mapping relationship, so as to fuse the two dimensional image onto the spherical image.

BACKGROUND OF THE DISCLOSURE 1. Field of the Disclosure

The present disclosure relates to the field of image processing, andmore particularly relates to a three dimensional image fusion method anddevice as well as a non-transitory computer-readable medium.

2. Description of the Related Art

Image fusion (for more information, seeen.wikipedia.org/wiki/Image_fusion) considers how to seamlessly combineone or more regions of interest in an image(s) into a target image. Sofar, a lot of research has been done in terms of two dimensional imagefusion. These kinds of research mainly focus on eliminating the colordifferences on the fused boundaries in a fused two dimensional image andadjusting the color of the fused two dimensional image to match a targetscene, so as to obtain a good image fusion result.

Compared to conventional two dimensional images, a three dimensionalimage or video may bring a more vivid visual experience to people. As aresult, the generation and editing of a three dimensional image or videohave become a very important research topic in recent years, whichinclude the studies on three dimensional image fusion. However, inconventional techniques, when fusing a two dimensional image or videoonto a three dimensional image such as a spherical image or the like,there are always discontinuous regions on the fused boundaries in thefused three dimensional image, so that it is impossible to acquire asatisfied image fusion result.

SUMMARY OF THE DISCLOSURE

In order to solve the above-described technical problem, the presentdisclosure provides a three dimensional image fusion method and device.

According to a first aspect of the present disclosure, a threedimensional image fusion method is provided which includes steps ofobtaining a spherical image and a two dimensional image to be fused ontothe spherical image; projecting the two dimensional image onto a planarsurface predetermined in a spherical coordinate system where thespherical image is located, so as to acquire a projected image which hasa predetermined number of feature points on a spherical surfacecorresponding to the spherical image; determining multiple mappingpoints on the spherical surface which correspond to the predeterminednumber of feature points, letting the predetermined number of featurepoints superpose on the multiple mapping points, respectively, andcarrying out spatial coordinate conversion, so as to attain a convertedimage; and mapping multiple points except the predetermined number offeature points in the converted image onto the spherical image based ona predetermined mapping relationship so as to fuse the two dimensionalimage onto the spherical image, so that a fused spherical image isprocured.

According to a second aspect of the present disclosure, a threedimensional image fusion device is provided which includes an imageobtaining part configured to obtain a spherical image and a twodimensional image to be fused onto the spherical image; an imageprojection part configured to project the two dimensional image onto aplanar surface predetermined in a spherical coordinate system where thespherical image is located, so as to acquire a projected image which hasa predetermined number of feature points on a spherical surfacecorresponding to the spherical image; a coordinate conversion partconfigured to determine multiple mapping points on the spherical surfacewhich correspond to the predetermined number of feature points, let thepredetermined number of feature points superpose on the multiple mappingpoints, respectively, and carrying out spatial coordinate conversion, soas to attain a converted image; and an image fusion part configured tomap multiple points except the predetermined number of feature points inthe converted image onto the spherical image based on a predeterminedmapping relationship so as to fuse the two dimensional image onto thespherical image, so that a fused spherical image is procured.

According to a third aspect of the present disclosure, another threedimensional image fusion device is provided which includes a processor;and a storage connected to the processor, storing computer-executableinstructions for execution by the processor. The computer-executableinstructions, when executed, cause the processor to implement the threedimensional image fusion method described above.

According to a fourth aspect of the present disclosure, a non-transitorycomputer-readable medium is provided which stores computer-executableinstructions for execution by a processing system. Thecomputer-executable instructions, when executed, cause the processingsystem to carry out the three dimensional fusion method set forth above.

As a result, by utilizing the three dimensional image fusion method orthe three dimensional image fusion devices, it is possible to seamlesslyfuse a two dimensional image or video onto a spherical image, i.e.,there are not any discontinuous regions or gaps on the fused boundariesin the fused spherical image, so that a satisfied image fusion resultmay be acquired. In addition, the process of this kind of image fusionis simple, and the practicality of this type of image fusion is strong.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example of a planar image which is captured by apanoramic camera and used for attaining a spherical image;

FIG. 2 is a flowchart of a three dimensional image fusion methodaccording to a first embodiment of the present disclosure;

FIG. 3 illustrates a two dimensional image to be fused onto a sphericalimage;

FIG. 4 illustrates a process of longitude and latitude based mapping(projection);

FIG. 5 illustrates a process of selecting the bottom surface of aspherical segment in a spherical coordinate system;

FIG. 6 illustrates a rectangle inscribed in the bottom surface of aspherical segment, onto which the two dimensional image shown in FIG. 3is projected;

FIG. 7 illustrates a process of projecting the two dimensional shown inFIG. 3 onto the spherical coordinate system so as to obtain a projectedimage;

FIG. 8 illustrates a process of conducting spatial coordinate conversionwith respect to the projected image presented in FIG. 7 so as to acquirea converted image;

FIG. 9 illustrates a process of mapping non-feature points in theconverted image shown in FIG. 8 onto the related spherical image;

FIG. 10 illustrates a process of performing longitude and latitude basedmapping (projection) on a fused spherical image so as to acquire asecond planar image;

FIG. 11 is a block diagram of a three dimensional image fusion deviceaccording to a second embodiment of the present disclosure; and

FIG. 12 is a block diagram of another three dimensional image fusiondevice according to a third embodiment of the present disclosure.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In order to let a person skilled in the art better understand thepresent disclosure, hereinafter, the embodiments of the presentdisclosure will be concretely described with reference to the drawings.However, it should be noted that the same symbols, which are in thespecification and the drawings, stand for constructional elements havingbasically the same function and structure, and the repetition of theexplanations to the constructional elements is omitted.

FIG. 1 illustrates an example of a planar image (also called a “firstplanar image” hereinafter) which is captured by a panoramic camera andused for attaining a spherical image.

Generally speaking, when utilizing a panoramic camera, for example, afisheye camera to acquire a spherical image, image distortion may occurin the relating first planar image as shown in FIG. 1 due to the imagingprinciple of the panoramic camera. In this case, if a predetermined twodimensional image or video is fused onto a location (e.g., a region ofinterest) in the relating first planar image by means of a conventionaltwo dimensional image fusion approach, then there may existdiscontinuous regions or gaps on the fused boundaries in the fusedspherical image because of the image distortion occurred in the relatingfirst planar image. As a result, it is impossible to achieve a seamlessimage fusion effect.

First Embodiment

In this embodiment, a three dimensional image fusion method is provided.

FIG. 2 is a flowchart of a three dimensional image fusion methodaccording to this embodiment.

Here it should be noted that the three dimensional image fusion methodmay also apply to image fusion between a two dimensional video and aspherical image.

As shown in FIG. 2, the three dimensional image fusion method isinclusive of STEPS S201 to S204.

In STEP S201 of FIG. 2, a spherical image and a two dimensional imagewhich will be fused onto the spherical image are obtained.

The spherical image may be acquired by a panoramic camera. Inparticular, first a first planar image, which is an equirectangularimage, as presented in FIG. 1 is taken by the panoramic camera. Secondlongitude and latitude based mapping (projection) is performed on thefirst planar image so as to procure a spherical image. In thisembodiment, the visual angle of the spherical image may be 360 degrees(i.e., the whole spherical surface) or 180 degrees (i.e., ahemispherical surface), and may also be 320 degrees, 270 degrees, etc.

The two dimensional image may have a polygonal shape. Preferably, theshape of the two dimensional image is a triangle, rectangle, square, orregular polygon.

In STEP S202 of FIG. 2, the two dimensional image is projected onto aplanar surface (plane) predetermined in the spherical coordinate systemwhere the spherical image is located, so as to obtain a projected image.The projected image has a predetermined number of feature points on thespherical surface corresponding to the spherical image.

Particularly, this step includes sub steps of acquiring a point on thespherical surface corresponding to the spherical image; attaining thebottom surface of a spherical segment, the distance between the bottomsurface and the point on the spherical surface being a predeterminedone; and projecting the two dimensional image onto the bottom surface ofthe spherical segment so as to obtain a projected image which isinscribed in the bottom surface (i.e., a circle) of the sphericalsegment.

In an example, it is possible to project, according to the size of thetwo dimensional image itself, the two dimensional image onto the bottomsurface of the spherical segment, so that the projected image isinscribed in the bottom surface of the spherical segment. Of course, itis also possible to enlarge or reduce the two dimensional image, andproject, in linear and equal proportion, the enlarged or reduced twodimensional image onto the bottom surface of the spherical segment, sothat the projected image is inscribed in the bottom surface of thespherical segment. The predetermined number of feature points that theprojected image has may be the intersections of the projected image andthe bottom surface of the spherical segment, i.e., the corner points ofthe projected image which are on the spherical surface corresponding tothe spherical image. Alternatively, the projected image may not beinscribed in the bottom surface of the spherical segment. In this case,it is possible to select a part of the edge points of the projectedimage, some points on a symmetric axis in the projected image, or aplurality of points on the boundaries of patterns in the projected imageto serves as the predetermined number of feature points of the projectedimage.

In STEP S203 of FIG. 2, multiple mapping points on the spherical surfacecorresponding to the spherical image are designated which correspond tothe predetermined number of feature points of the projected image. Andthen, the predetermined number of feature points are superposed on thecorresponding mapping points, and spatial coordinate conversion iscarried out, so that a converted image is attained.

In this step, the multiple mapping points on the spherical surfacecorresponding to the spherical image, corresponding to the predeterminednumber of feature points of the projected image may be determined inadvance, or may also be chosen according to a predetermined rule. Thepositions of the multiple mapping points may be dependent on thelocation of a predetermined region of interest on the spherical image,onto which the two dimensional image will be projected, for example, thelocation of a display screen in the spherical image. After the positionsof the multiple mapping points are designated, it is possible to create,on the basis of the positions of the predetermined number of featurepoints and the corresponding mapping points, a perspectivetransformation matrix, i.e., a homography matrix (for more information,see en.wikipedia.org/wiki/Homography_(computer_vision)), and acquire theconverted image on the grounds of the projected image and the homographymatrix. Here it should be noted that the predetermined number of featurepoints and the corresponding mapping points may be partly or entirelysuperimposed, or may also not be superimposed at all.

In particular, the predetermined number of feature points of theprojected image may be dragged to the corresponding mapping points onthe spherical surface corresponding to the spherical image so as to leteach feature point superpose on the corresponding mapping point. In thisprocess, it is possible to conduct triangular surface conversion withrespect to the projected image (i.e., divide the projected image intomultiple triangular surfaces), and create, on the basis of the multipletriangular surfaces, a homography matrix between time points before andafter being dragged so as to obtain a mapping matrix between theprojected image and the converted image, thereby generating a convertedimage.

In STEP S204 of FIG. 2, multiple points (also called “non-featurepoints” hereinafter) except the predetermined number of feature pointsin the converted image are mapped onto the spherical image according toa predetermined mapping relationship, so as to let the two dimensionalimage be fused on the spherical image.

As described above, in STEP S203 of FIG. 2, all the predetermined numberof feature points on the converted image are located at the multiplemapping points on the spherical surface corresponding to the sphericalimage, respectively. That is, all the predetermined number of featurepoints in the converted image are on the spherical surface correspondingto the spherical image. The purpose of this step is to map the multiplenon-feature points in the converted image onto the spherical image.

Particularly, first the intersections of the spherical surfacecorresponding to the spherical image and a straight line, which passesthrough the spherical center of the spherical surface and a non-featurepoint on the converted image, are acquired. In general, the number ofthe acquired intersections are two; as such, it is possible todetermine, according to a region needing to be mapped or otherconfigurations, one of the acquired intersections as an intersection formapping. Second the non-feature point on the converted image is mappedonto the intersection for mapping, and the corresponding original pointon the spherical image is replaced at the same time. After this kind ofprocess is conducted with respect to all the non-feature points on theconverted image, the two dimensional image is fused on the sphericalimage. In this way, it is possible to produce a new spherical image,i.e., a fused spherical image.

In an example, the three dimensional image fusion method may furtherinclude a step of carrying out longitude and latitude based mapping(projection) in regard to the fused spherical image so as to procure asecond planar image which is also an equirectangular image.

Additionally, in a case where a user wants to change the location on thespherical image onto which the two dimensional image will be fused(e.g., the location of a display screen in the spherical image) on thebasis of the above, it is possible to set multiple mapping points on thespherical surface corresponding to the spherical image again, acquire anew converted image, and map the new converted image onto the sphericalimage in the same way.

As a result, by taking advantage of the three dimensional image fusionmethod, it is possible to seamlessly fuse a two dimensional image onto aspherical image, i.e., there are not any discontinuous regions or gapson the fused boundaries in the fused spherical image, so that asatisfied image fusion result may be acquired. In addition, the processof this kind of image fusion is simple, the practicality of this type ofimage fusion is strong, and the requirements of different visual anglesand fusion locations can be met.

In what follows, an example is given for concretely describing the threedimensional image fusion method by referring to FIGS. 3 to 10.

FIG. 3 illustrates a two dimensional image to be fused onto thespherical image corresponding to the first planar image shown in FIG. 1.

As described above, the spherical image corresponding to the firstplanar image shown in FIG. 1 may be generated by carrying out longitudeand latitude based mapping (projection) with respect to the first planarimage.

FIG. 4 illustrates a process of longitude and latitude based mapping(projection).

As presented in the left image (i.e., a first planar image) in FIG. 4,(u,v) refers to the coordinates of a pixel point P therein, and thewidth and height of the first planar image are W and H, respectively.After the first planar image is projected onto a spherical coordinatesystem (r,θ, φ), as shown in the right image in this drawing, the radiusof the sphere acquired is R, and the spherical coordinates of the pixelpoint P after projection are (x,y,z). Here,

${\varphi = {\frac{u}{W} \times 2\pi}};$${\theta = {\frac{v}{H} \times 2\pi}};$ x = R × sin  θ × cos  φ;y = R × cos  θ; and z = R × sin  θ × sin  φ.

FIG. 5 illustrates a process of choosing the bottom surface of aspherical segment in the spherical coordinate system.

FIG. 6 illustrates a rectangle inscribed in the bottom surface of thespherical segment onto which the two dimensional image is projected.

As shown in FIG. 5, a point P(x,y,z) on the spherical surfacecorresponding to the spherical image is obtained in the sphericalcoordinate system, and then, the bottom surface of a spherical segmentis selected whose radius is r and the distance between which and thepoint P(x,y,z) is h, so that the two dimensional image is projected, inlinear and equal proportion, onto the bottom surface of the sphericalsegment so as to acquire a rectangle as a projected image inscribedtherein whose center is P₀(x₀,y₀,z₀), as shown in FIG. 6. Here, the fourcorner points P₁(x₁,y₁,z₁), P₂(x₂,y₂,z₂), P₃(x₃,y₃,z₃), and P₄(x₄,y₄,z₄)of the inscribed rectangle presented in FIG. 6 are the predeterminednumber of feature points of the projected image as depicted above.

FIG. 7 illustrates a process of projecting the two dimensional imageshown in FIG. 3 onto the spherical coordinate system.

If P₁(x₁,y₁,z₁) shown in FIG. 6 is taken as an example, then

${\varphi = {\tan^{- 1}\frac{z}{x}}};$${\theta = {\tan^{- 1}\left( \frac{\sqrt{x^{2} + z^{2}}}{y} \right)}};$x₀ = (R − h) × sin  θ × cos  φ; y₀ = (R − h) × cos  θ;z₀ = (R − h) × sin  θ × sin  φ;${r = {\sqrt{4{h\left( {{2R} - h} \right)}} = \sqrt{\left( {x_{1} - x_{0}} \right)^{2} + \left( {y_{1} - y_{0}} \right)^{2} + \left( {z_{1} - z_{0}} \right)^{2}}}},{and}$$R = {\sqrt{x_{1}^{2} + y_{1}^{2} + z_{1}^{2}}.}$

Of course, for each of P₂(x₂,y₂,z₂), P₃(x₃,y₃,z₃), and P₄(x₄,y₄,z₄)presented in FIG. 6, it is the same.

FIG. 8 illustrates a process of conduct spatial coordinate conversionwith respect to the projected image so as to acquire a converted image.

As presented in FIG. 8, it is possible to designate four mapping pointsP₁′, P₂′, P₃′, and P₄′ on the spherical surface corresponding to thespherical image, corresponding to the four feature points P₁, P₂, P₃,and P₄ of the projected image on the grounds of a predetermined locationin the spherical surface onto which the two dimensional image needs tobe fused, and then, drag the four feature points P₁, P₂, P₃, and P₄ tothe positions of the four mapping points P₁′, P₂′, P₃′, and P₄′,respectively. In this dragging process, the projected image is deformedso as to attain a converted image.

Particularly, it is possible to respectively drag the four featurepoints P₁, P₂, P₃, and P₄ to the mapping points P₁′, P₂′, P₃′, and P₄′,so as to let each feature point superpose on the corresponding mappingpoint. In this dragging process, it is possible to conduct triangularsurface conversion with respect to the projected image (i.e., divide theprojected image into multiple triangular surfaces), and then, create, onthe basis of the multiple triangular surfaces, a homography matrixbetween time points before and after being dragged so as to obtain amapping matrix between the projected image and the converted image,thereby generating the converted image. It is thus clear that thespatial shape of the converted image and the positions of the respectivepoints therein rely on the joint effects of dragging the four featurepoints P₁, P₂, P₃, and P₄.

FIG. 9 illustrates a process of mapping multiple points except the fourfeature points in the converted image onto the spherical image.

Since all the four feature points acquired in the converted imagepresented in FIG. 8 are located on the spherical surface correspondingto the spherical image, the multiple points (i.e., non-feature points)except the four feature points P₁, P₂, P₃, and P₄ in the converted imagemay be mapped onto the spherical image, as shown in FIG. 9.

In particular, it is possible to obtain a straight line passing throughthe spherical center of the spherical surface corresponding to thespherical image and a non-feature point q(x_(q),y_(q),z_(q)) on theconverted image, and then, attain an intersectionq′(x_(q)′,y_(q)′,z_(q)′) of the straight line and the spherical surface,so as to map the non-feature point q onto the intersection q′ andreplace the corresponding original point on the spherical image. Sincethe straight line passing through the spherical center of the sphericalsurface and the non-feature point q on the converted image usually hastwo intersections with the spherical surface, it is possible to selectthe intersection q′ for mapping on the grounds of the predeterminedlocation on the spherical surface onto which the two dimensional imageneeds to be fused.

Here, as shown in FIG. 9,

${\varphi = {\tan^{- 1}\frac{z_{q}}{x_{q}}}};$${\theta = {\tan^{- 1}\left( \frac{\sqrt{x_{q}^{2} + z_{q\;}^{2}}}{y_{d}} \right)}};$$r_{0} = \sqrt{R^{2} - y^{2}}$ x_(q)^(′) = r × cos  φ;y_(q)^(′) = R × cos  θ; and z_(q)^(′) = r × sin  φ.

After all the non-feature points on the converted image are mapped tothe spherical image, and the corresponding points on the spherical imageare replaced, the two-dimensional image is fused onto the sphericalimage is finished. That is, a new spherical image, i.e., a fusedspherical image is procured.

Finally, it is also possible to carrying out longitude and latitudebased mapping (projection) (as shown in FIG. 4) with respect to thefused spherical image so as to attain a second planar image which isalso an equirectangular image, as shown in FIG. 10. Here, FIG. 10illustrates a process of performing longitude and latitude based mapping(projection) on a spherical image after image fusion so as to acquire asecond planar image.

Second Embodiment

A three dimensional image fusion device is given in this embodiment.

FIG. 11 is a block diagram of a three dimensional image fusion device1100 according to this embodiment.

As shown in FIG. 11, the three dimensional image fusion device 1100 isinclusive of an image obtainment part 1110, an image projection part1120, a coordinate conversion part 1130, and an image fusion part 1140.Aside from these parts, the dimensional image fusion device 1100 mayalso contain other parts, of course. However, since the other parts arenot closely relating to the embodiments of the present disclosure, thedescriptions about them are omitted here for the sake of convenience.

The image obtainment part 1110 is configured to conduct STEP S201 inFIG. 2, i.e., to obtain a spherical image and a two dimensional imagewhich will be fused onto the spherical image. The spherical image may begenerated on the basis of a first planar image which is anequirectangular image.

The image projection part 1120 is configured to perform STEP S202 inFIG. 2, i.e., to project the two dimensional image onto a planar surfacepredetermined in the spherical coordinate system where the sphericalimage is located, so as to obtain a projected image. The projected imagehas a predetermined number of feature points on the spherical surfacecorresponding to the spherical image.

The coordinate conversion part 1130 is configured to carry out STEP S203in FIG. 2, i.e., to designate multiple mapping points on the sphericalsurface corresponding to the spherical image which correspond to thepredetermined number of feature points of the projected image, let thepredetermined number of feature points superpose on the correspondingmapping points, and conduct spatial coordinate conversion so as toattain a converted image.

The image fusion part 1104 is configured to execute STEP S204 in FIG. 2,i.e., to map multiple points except the predetermined number of featurepoints on the converted image onto the spherical image according to apredetermined mapping relationship so as to let the two dimensionalimage be fused on the spherical image. In this way, it is possible toacquire a fused spherical image.

Here it should be noted that since STEPS S201 to S204 of FIG. 2 havebeen detailed in the first embodiment, the descriptions regarding themare omitted here for the sake of convenience.

Moreover, the three dimensional image fusion device 1100 may furthercontain an image acquirement part which is not presented in thedrawings. The image acquirement part is configured to perform longitudeand latitude based mapping (projection) on the fused spherical image soas to procure a second planar image which is also an equirectangularimage.

Furthermore, if a user wants to change the location on the sphericalimage onto which the two dimensional image will be fused (e.g., thelocation of a display screen in the spherical image) in light of theimage fusion process set forth above, it is also possible to use thecoordinate conversion part 1103 to determine multiple mapping points onthe spherical surface corresponding to the spherical image again andacquire a new converted image, and then, utilize the image fusion part1104 to map the new converted image onto the spherical image in the sameway.

As a result, by making use of the three dimensional image fusion device1100, it is possible to seamlessly fuse a two dimensional image onto aspherical image, i.e., there are not any discontinuous regions or gapson the fused boundaries in the fused spherical image, so that asatisfied image fusion result may be acquired. In addition, the processof this type of image fusion is simple, the practicality of this kind ofimage fusion is strong, and the requirements of different visual anglesand fusion locations can be met.

Third Embodiment

In this embodiment, another three dimensional image fusion device isprovided.

FIG. 12 is a block diagram of a three dimensional image fusion device1200 according to this embodiment.

Here it should be noted that the three dimensional image fusion device1200 may be a computer or server.

As presented in FIG. 12, the three dimensional image fusion device 1200contains at least one processor 1210 and a storage 1220. Of course, itis also possible to include other elements such as a panoramic cameraand an output unit (not shown in the drawings) for good measure. Thesekinds of elements may be connected to each other by way of a bus system,for example.

The storage 1220 is configured to store computer-executable instructions(i.e. an application program) for execution by the processor 1210 andintermediate data during a calculation process of the processor 1210.The computer-executable instructions, when executed, may cause theprocessor 1210 to carry out the three dimensional image fusion methodaccording to the first embodiment.

As a result, by utilizing the three dimensional image fusion device1200, it is possible to seamlessly fuse a two dimensional image onto aspherical image, i.e., there are not any discontinuous regions or gapson the fused boundaries in the fused spherical image, so that asatisfied image fusion result may be acquired. In addition, the processof this kind of image fusion is simple, the practicality of this type ofimage fusion is strong, and the requirements of different visual anglesand image fusion locations can be met.

Here it should be noted that the embodiments of the present disclosuremay be implemented in any convenient form, for example, using dedicatedhardware or a mixture of dedicated hardware and software. Theembodiments of the present disclosure may be implemented as computersoftware executed by one or more networked processing apparatuses. Thenetwork may comprise any conventional terrestrial or wirelesscommunications network, such as the Internet. The processing apparatusesmay comprise any suitably programmed apparatuses such as ageneral-purpose computer, a personal digital assistant, a mobiletelephone (such as a WAP or 3G-compliant phone) and so on. Since theembodiments of the present disclosure can be implemented as software,each and every aspect of the present disclosure thus encompassescomputer software implementable on a programmable device.

The computer software may be provided to the programmable device usingany storage medium for storing processor-readable code such as a floppydisk, a hard disk, a CD ROM, a magnetic tape device or a solid statememory device.

The hardware platform includes any desired hardware resources including,for example, a central processing unit (CPU), a random access memory(RAM), and a hard disk drive (HDD). The CPU may include processors ofany desired type and number. The RAM may include any desired volatile ornonvolatile memory. The HDD may include any desired nonvolatile memorycapable of storing a large amount of data. The hardware resources mayfurther include an input device, an output device, and a network devicein accordance with the type of the apparatus. The HDD may be providedexternal to the apparatus as long as the HDD is accessible from theapparatus. In this case, the CPU, for example, the cache memory of theCPU, and the RAM may operate as a physical memory or a primary memory ofthe apparatus, while the HDD may operate as a secondary memory of theapparatus.

While the present disclosure is described with reference to the specificembodiments chosen for purpose of illustration, it should be apparentthat the present disclosure is not limited to these embodiments, butnumerous modifications could be made thereto by a person skilled in theart without departing from the basic concept and technical scope of thepresent disclosure.

The present application is based on and claims the benefit of priorityof Chinese Patent Application No. 201710099062.6 filed on Feb. 23, 2017,the entire contents of which are hereby incorporated by reference.

What is claimed is:
 1. A three dimensional image fusion methodcomprising: obtaining a spherical image and a two dimensional image tobe fused onto the spherical image; projecting the two dimensional imageonto a planar surface predetermined in a spherical coordinate systemwhere the spherical image is located, so as to acquire a projected imagewhich has a predetermined number of feature points on a sphericalsurface corresponding to the spherical image; determining multiplemapping points on the spherical surface which correspond to thepredetermined number of feature points, letting the predetermined numberof feature points superpose on the multiple mapping points,respectively, and carrying out spatial coordinate conversion, so as toattain a converted image; and mapping multiple points except thepredetermined number of feature points in the converted image onto thespherical image based on a predetermined mapping relationship so as tofuse the two dimensional image onto the spherical image, so that a fusedspherical image is procured.
 2. The three dimensional image fusionmethod according to claim 1, wherein, the obtaining a spherical imageincludes acquiring a first planar image and performing longitude andlatitude based projection on the first planar image, so as to obtain thespherical image.
 3. The three dimensional image fusion method accordingto claim 1, wherein, the projecting the two dimensional image onto aplanar surface predetermined in a spherical coordinate system where thespherical image is located, so as to acquire a projected image includesobtaining a point on the spherical surface corresponding to thespherical image; attaining a bottom surface of a spherical segment, adistance from the bottom surface to the point on the spherical surfacecorresponding to the spherical image being a predetermined one; andprojecting the two dimensional image onto the bottom surface of thespherical segment so as to acquire the projected image which isinscribed in the bottom surface of the spherical segment.
 4. The threedimensional image fusion method according to claim 3, wherein, theprojecting the two dimensional image onto the bottom surface of thespherical segment includes projecting the two dimensional image on thebottom surface of the spherical segment in linear and equal proportion.5. The three dimensional image fusion method according to claim 1,wherein, the predetermined number of feature points that the projectedimage has are edge points and/or corner points of the projected image.6. The three dimensional image fusion method according to claim 1,wherein, the two dimensional image has a regular polygonal shape.
 7. Thethree dimensional image fusion method according to claim 1, wherein, theletting the predetermined number of feature points superpose on themultiple mapping points, respectively, and carrying out spatialcoordinate conversion, so as to attain a converted image includescreating a homography matrix based on positions of the predeterminednumber of feature points and the multiple mapping points, and attainingthe converted image based on the two dimensional image and thehomography matrix.
 8. The three dimensional image fusion methodaccording to claim 1, wherein, the mapping multiple points except thepredetermined number of feature points in the converted image onto thespherical image based on a predetermined mapping relationship includesacquiring an intersection of the spherical surface and a straight linewhich passes through a spherical center of the spherical surfacecorresponding to the spherical image and each non-feature point on theconverted image; and mapping each non-feature point in the convertedimage onto the corresponding intersection, and replacing a correspondingoriginal point on the spherical image.
 9. The three dimensional imagefusion method according to claim 1, further comprising: carrying outlongitude and latitude based projection with respect to the fusedspherical image so as to attain a second planar surface.
 10. A threedimensional image fusion device comprising: a processor; and a storageconnected to the processor, storing computer-executable instructions forexecution by the processor, wherein, the computer-executableinstructions, when executed, cause the processor to implement the threedimensional image fusion method according to claim
 1. 11. Anon-transitory computer-readable medium having computer-executableinstructions for execution by a processing system, wherein, thecomputer-executable instructions, when executed, cause the processingsystem to carry out the three dimensional image fusion method accordingto claim
 1. 12. A three dimensional image fusion device comprising: animage obtaining part configured to obtain a spherical image and a twodimensional image to be fused onto the spherical image; an imageprojection part configured to project the two dimensional image onto aplanar surface predetermined in a spherical coordinate system where thespherical image is located, so as to acquire a projected image which hasa predetermined number of feature points on a spherical surfacecorresponding to the spherical image; a coordinate conversion partconfigured to determine multiple mapping points on the spherical surfacewhich correspond to the predetermined number of feature points, let thepredetermined number of feature points superpose on the multiple mappingpoints, respectively, and carrying out spatial coordinate conversion, soas to attain a converted image; and an image fusion part configured tomap multiple points except the predetermined number of feature points inthe converted image onto the spherical image based on a predeterminedmapping relationship so as to fuse the two dimensional image onto thespherical image, so that a fused spherical image is generated.